Performance Measures for Alcohol and Other Drug Services

Performance measures, which evaluate how well health care practitioners’ actions conform to practice guidelines, medical review criteria, or standards of quality, can be used to improve access to treatment and the quality of treatment for people with alcohol and other drug problems. This article examines different types of quality measures, how they fit within the continuum of care, and the types of data that can be used to arrive at these measures. The Washington Circle measures—identification, initiation of treatment, and treatment engagement—are a widely used set of performance measures.

P erformance measures--the methods or instruments used to evaluate the extent to which health care practitioners' actions conform to practice guidelines, medical review criteria, or standards of quality (AcademyHealth 2004)-can be valuable tools for improving access to treatment services and the quality of those services for people with alcohol and other drug (AOD) problems.
This article first examines three important aspects of the development and use of performance measures for AOD services: a continuum-of-care framework, the types of quality measures, and the types of data from which these measures can be derived. Choices the users (e.g., State agencies or providers) make in each of these areas are determined by the users' goals and the settings where the performance measure will be applied (Horgan and Garnick 2005). This article also describes the set of performance measures developed by the Washington Circle and widely adopted in 2003, explaining the development, testing, and current use of these measures.

Continuum-of-Care Framework
People who are experiencing various stages of AOD disorders have a variety of needs that require attention if they are to receive optimal care. Services can be seen as falling into the following stages along a continuum of care (McCorry et al. 2000): • Prevention/Education: Encompasses activities designed to raise the gen eral awareness of AOD disorders as a major issue affecting individuals, families, and society. It also includes activities designed to target high-risk people and groups for more focused interventions.
• Recognition: Focuses on identifying people with AOD disorders and includes activities such as screening, assessment, and referral to treatment.
• Treatment: Includes a broad range of services associated with treatment and rehabilitation, such as counsel ing, social services, medications, testing, and coordination with other treatment resources.
• Maintenance: Concentrates on sus taining positive outcomes over the longer term; includes self-manage ment and peer support strategies intended to sustain post-treatment abstinence or reductions in use, improvements in role functioning and well-being, and lifestyle changes.
Performance measures should be developed and used in each of these stages of care, because excellence in one domain at the expense of the others will result in less than optimal patient care. For example, if a health plan pro vides comprehensive AOD treatment services but does not screen people enrolled in the plan to identify those with AOD abuse or dependence (i.e., recognition), it is failing to provide treatment to people who need it.

Types of Quality Measures
Health care quality is defined by the Institute of Medicine (2001) as "the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current profes sional knowledge" (p. 44). Building on the classic framework proposed by Donabedian (1980), researchers have developed five categories of quality measures: structure, process, outcomes, access, and patient experience.

Structural Measures
Structural quality measures refer to the features of a health care organization that determine its capacity to provide care, such as the existence of an elec tronic records system or the ratio of AOD treatment providers to clients.

Process Measures
Process measures are used to assess how well a health care service provided to a patient, or on a patient's behalf, adheres to recommendations for clinical prac tice. These recommendations are based on research evidence or consensus (i.e., the views of experts when the research evidence is lacking or inconclusive) regarding the probability that provid ing particular services will achieve the desired outcomes. Generally, process measures are expressed as rates, with the denominator defining a population that is of interest because of its demo graphic and clinical condition, and the numerator defining the subgroup receiving specific services.
Process measures are particularly important in the development of per formance measures because they can be used to identify specific areas of care that may need improvement. For exam ple, if clients are not returning after an initial service, outreach efforts could be mounted in an effort to retain clients in treatment. Moreover, the data to cal culate process measures often can be obtained as part of an administrative data system that includes information on clients' dates and types of treatment services. Recently, researchers have devel oped processes by which a range of stakeholders (e.g., clinicians, specialists in measurement) can work together to select core sets of process measures for common use based on analyses of how meaningful and feasible different process measures are (Hermann et al. 2004).

Outcome Measures
These measures generally are used to evaluate the state of a patient's health resulting from the health care services and interventions received. In general, outcomes can be considered both in terms of patient functioning and in terms of categories of symptom severity related to the patient's clinical problem. For AOD disorders, health outcomes can be expanded to include four areas: sustained reductions in AOD use, improvements in personal health, sus tained improvements in functioning (e.g., employment), and sustained reductions in threats to public health and safety (McLellan et al. 2005).
Attributing improved patient out comes to providers' specific actions can be difficult because outcome measures of quality reflect the cumulative impact of multiple factors such as the timeli ness of services and the appropriateness of the type or amount of services for a person's substance abuse or dependence problem. Outcomes also are influenced by factors that depend on the patients themselves and are outside the health care system, such as the choices the patients make (e.g., to remain in treat ment for the full course of recommended services); their living situation, income, and employment; or whether the patients have other medical conditions.

Access Measures
Access measures assess the extent to which a person who needs care and services is able to receive them. This type of measure is particularly impor tant for addictive disorders because of the large gap between those who have serious problems and those who actu ally use any services.

Patient Experience
These measures are aggregated from patients' reports about their observa tions of and participation in AOD treatment.

Data Sources for Performance Measures
Sources of data for performance mea sures include medical records, patient surveys, and administrative data used to pay bills or to manage care in health plans, government programs, and State substance abuse systems.

Medical Records
Medical records provide detailed infor mation on clinicians' observations, impressions, thought processes, and referrals to other providers; and on patients' tests, results, medications, and reactions to clinicians' advice. A diffi culty with using medical records as a data source, however, is that patients often receive services in multiple sites in the specialty substance abuse, mental health, and general medical sectors, and each of these sites generates its own set of medical records. Unfortunately, it is difficult for various segments of the health care system or for behavioral health treatment facilities to share elec tronic medical records (McLellan et al. 2003), primarily because the various providers do not share a vocabulary or other established industry standards that would guide the development of effec tive computerized medical record data systems. Thus, medical records are not commonly used for performance mea sures because of the cost and complexity of compiling the necessary data from a wide range of medical record formats.

Patient/Enrollee Surveys
Surveys can capture information on how patients view their treatment expe rience, including their satisfaction with care and its various treatment compo nents or events (e.g., being offered information about a medication's side effects) and their perceptions about the outcomes of care. Performance mea sures based on patient experience spe cific to addiction treatment still are being developed, but two major efforts are worthy of mention.
The Experience of Care and Health Outcomes (ECHO) Survey (CAHPS Survey Users Network 2004) includes questions on a wide range of issues, such as access to care, experience of providers' communication and listening skills, satisfaction with care, and perceived benefits of treatment. From 2002 through 2004, the National Committee for Quality Assurance (NCQA) adopted a version of this survey for commercial managed behavioral health care organiza tions, though not for general health plans (CAHPS Survey Users Network 2004).
The Modular Survey Initiative, spon sored by the Substance Abuse and Mental Health Services Administration (SAMHSA), is designed to develop and promote a set of consumer perceptionof-care survey tools that can be widely used across settings and populations (Doucette 2004;Merrick 2004;. The modular survey consists of a core set of items addressing quality of care, access to services, and perceived improvement as a result of having received services. Additional modules addressing the quality of the helping alliance, motivation to change, and relapse/recovery status are under devel opment. The modular framework increases the flexibility of the survey in that some survey modules are appropri ate for the full population range and others are specific to a particular type of service (e.g., AOD disorders or mental health), age group (e.g., youth or adults), service settings (mental health/AOD), and delivery/financing vehicles (i.e., public or private, insured or unin sured). Table 1 shows examples of the survey's domains and survey items within those domains.

Administrative Data
In the course of providing and paying for care, insurers and related organiza tions generate administrative data on the characteristics of the population they serve as well as the utilization of and charges for services, often at an individual-user level. Even without full standardization of data collection, these data usually contain certain key ele ments, such as the following (Garnick et al. 2002a): • Amount billed and amount reim bursed, often separately by service • Type of service • Number of units (e.g., days of service) • Location of service • Diagnosis and procedure codes for clinical services.
Patients receiving AOD-related ser vices can be identified and their care tracked over time and across settings via administrative data. Large amounts of data are available, information already is collected by organizations that are familiar with its use, and data usually are collected with common data elements and formats.
Insurance claims data, one type of administrative data, have been used for measuring the quality of AOD services in a variety of settings, including com mercial health plans and the Federal Government's Medicare and Medicaid programs. In addition, systems such as staff-model managed care plans, which directly manage and pay physicians, as well as State-run specialty hospitals and Department of Veterans' Affairs (VA) facilities, deliver care through facilities they own and providers they employ. Most of these systems generate encounterlevel records that have dates and descrip tions of services, such as are included in claims.
Data generated by publicly funded mental health and AOD-treatment systems administered through State departments of public health, mental health, or substance abuse could be useful for performance measurement. Across the 50 States, the availability of data to support performance measure ment varies widely, however. Some States are leaders in data systems and have detailed data on date of service, types of services, and diagnoses for spe cific clients, but many still report only aggregate service-use statistics or the start and end dates of episodes of care without specific detail on dates for all services. More States are moving toward designing systems to accumu late the level of detail required for many performance measures.
Despite the many advantages of administrative data, this data source also has quality problems. AOD diag noses may not be coded accurately or completely because of issues of insura bility, confidentiality, stigma, or even lack of space on the claims forms to record multiple diagnoses. Furthermore, codes for procedures specific to the treatment of addictions are not included in the Current Procedural Terminology (CPT), which is the most widely used coding system for procedures. (The CPT is published by the American Medical Association and updated annually.) A common coding practice, for example, is to indicate a substance use disorder diagnosis along with a generic psychi atric procedure code that does not dif ferentiate between services for mental health problems and substance use disor ders. This makes calculating perfor mance measures much more complex.
The Health Insurance Portability and Accountability Act (HIPAA) con tains electronic transaction regulations requiring a national procedure code set. These requirements, which were imple mented as of 2003, could solve the problem posed by the lack of treatment codes for services related to substance use disorders. Not only has HIPAA standardized codes for procedures and durations of treatment, but the services code set has been expanded to more precisely reflect the wide range of exist ing types of services for substance use disorders (Centers for Medicare and Medicaid Services 2005;SAMHSA 2004). This code set, Level II of the Healthcare Common Procedural Coding System (HCPCS), is used to identify items and services, including behavioral services that are not included in the CPT codes. See the textbox for a list of selected items from this expanded code set.

Developing Performance Measures: The Washington Circle Example
In 1998, the Center for Substance Abuse Treatment convened the Washington Circle, a multidisciplinary group of providers, researchers, managed care representatives, and public policymakers, to address the need for performance measures for people with AOD disor ders (McCorry et al. 2000). Building on the foundation laid by several other groups in both the public and private sectors, this group outlined a set of seven performance measures that can apply to all four stages of the contin uum of care--prevention/education, recognition, treatment, and maintenance.
During 1999, the Washington Circle selected three of these measures, defined in detail below, for initial focus. From 1999 through 2002, the group wrote technical specifications and car ried out pilot testing (Garnick et al. 2002b). The measures were applied first in commercial managed care plans that had enrolled populations and administrative datasets which used common definitions and contained detailed data on services and dates of enrollment. The Washington Circle

Identification of AOD Services.
This measure captures the percentage of a health plan's enrollees that receive any addiction-related services; it falls into the recognition stage of the continuum of care (described above). If health plans actively screen their enrollees for AOD problems, and if providers code substance abuse or dependence diag noses more accurately, the health plan likely will score higher on this measure. Specifically, this measure reports the percentage of health plan members with at least one claim containing a diagnosis of AOD abuse or depen dence and services related to the treat ment of these diagnoses during the measurement year.

Initiation of Treatment.
The initiation of treatment measure captures informa tion on patients with substance abuse or dependence diagnoses who return for additional services. This is a process measure that falls into the treatment domain on the continuum of care. Specifically, it is defined as the percent age of adults diagnosed with a new episode of AOD abuse or dependence and receiving a related service who either (1) initiate treatment through an inpatient AOD admission or (2) have an initial outpatient service for AOD abuse or dependence and receive any additional AOD services within 14 days.
Treatment Engagement. This is an intermediate step between initially accessing care (in the first visit) and completing a full course of treatment. This measure is defined as the percent age of adults diagnosed with AOD abuse or dependence who receive two additional AOD treatments within 30 days after initiating treatment.

Key Decisions in Developing the Washington Circle Measures
Defining a service that begins a new episode of care is essential for many performance measures (Hornbrook et al. 1985). Both the initiation and engagement measures rely on finding people with new episodes of AOD ser vices; this is the population of interest, and they are represented by the denom inator. To qualify as a new episode, there must be a period of 60 days, referred to as a "negative diagnosis history," during which the person had no claims or encounters associated with any diagno sis of AOD abuse or dependence. This period without any claims also is called a "clean" period in the literature on episodes of care. An analysis of adults who were commercially insured showed that extending this negative diagnosis period from 60 to 90 days reduced the number of new AOD episodes, as expected, but the initiation and engage ment rates did not change markedly (Horgan et al. 2004). Thus, this analytic decision reflects a trade-off between having a larger number of people included in the measure and having more confi dence that the people selected actually are beginning new episodes of treatment.
In developing these measures, the Washington Circle gave special consid eration to some services. In particular, the group decided that detoxification and emergency room care could mark the beginning of a treatment episode but were not appropriate for continuing care, as these services do not by them selves constitute treatment for AOD disorders. In addition, participation in Alcoholics Anonymous and other com munity support fellowships was not included in the measures, in part because data are not collected on participation. Moreover, although these groups clearly are important for the continuation of treatment-initiated sobriety and social function, by their own charters they are neither structured to provide the pro fessional services that so often are needed to stabilize psychologically and emo tionally unstable patients, nor designed to address the complications of addiction which can interfere with behavioral change (McCorry et al. 2000).

Applications of Washington Circle Performance Measures
Several groups have calculated the three performance measures discussed here--identification, initiation, and engagement--throughout the testing and development process and in realworld applications. Although these measures were developed for commer cial health plans, they also have been used with a State data system, VA data, and Medicaid data. The applications described here share common themes. They all demonstrate the use of data that currently are collected for adminis trative purposes to monitor quality, and they all build on the performance measures for AOD services developed by the Washington Circle and adopted by NCQA in 2004. . Table 2A compares AOD identification, initiation of treatment, and treatment engagement rates for commercial health plans using three data sources:

Commercial Health Plans
• The Washington Circle pilot test using data from six managed care plans (the identities of which are confidential).
• 2001 data from Marketscan ® , a large database maintained by Medstat, which consists of claims submitted from private health insurance com panies and managed care vendors across the United States for about a quarter million adults age 18 and older.
• Data compiled by NCQA for the first year that the measures were used by health plans nationwide.
These measures also were used to compare the periods before and after the Federal Employees Health Benefits Program implemented parity for six health plans (data not shown) (Northrop Grumman Information Technology 2004). Although the performance rates vary considerably, some common themes emerge. The identification rates are all under 1.5 percent, which is consider ably less than the approximately 5 percent of commercially insured adults estimated to need treatment for substance abuse or dependence. Initiation rates ranged from 26 to 46 percent, and engagement rates ranged from 14 to 49 percent. Table 2B compares the same performance measures shown in Table 2A using three data sources from the public sector: VA data (Harris et al. 2005), data from the Oklahoma Department of Mental Health and Substance Abuse Services (2005), and

Performance Measures for AOD Services
Medicaid data from more than 40 States (Tompkins and Reif 2005).
Because researchers are still collaborat ing with States to adapt the Washington Circle measures to populations served by public funds, the results shown in the table are intended to be illustrative. The public sector identification rates are higher than in the commercial health plans, ranging from 6.3 percent in the VA to 9 percent in the Oklahoma State system. Comparing rates for vari ous populations requires great caution, however, because types of data and data collection methods vary. For example, as a State mental health and substance abuse system, Oklahoma does not have an enrolled population, so administra tors elected to use adults with family incomes less than twice the Federal poverty level (i.e., under 200 percent of poverty) as an estimate of the popula tion their treatment system serves, and thus as the denominator for the identi fication rate. For initiation and engage ment rates, the Medicaid analysis includes services provided in both spe cialty and in general medical settings, whereas the Oklahoma results are lim ited to specialty settings.
Two important considerations in relation to using performance measures in both the public and private sectors are how to interpret the data and how to avoid setting up incentives that encour age a health plan to focus on one mea sure at the expense of others. In terms of interpretation, data on performance measures should be interpreted with care. For example, if a health plan has a high identification rate, it may be appropriately identifying people, but some of these people may be more dif ficult to motivate into treatment. Thus, the plan would show a lower engage ment rate. A low engagement rate could be explained by the health plan as resulting from barriers to continued care, such as limited referral networks or complex approval systems. In terms of incentives, if a health plan offers incentives for improving one particular performance measure (e.g., identifica tion), that measure may improve through increased screening, provider education, or outreach to enrollees, for example, but at the expense of the other two measures (e.g., treatment initiation and engagement).

Future Directions
Performance measures are potentially powerful tools for commercial health plans, State providers (through Medicaid programs and State-run treatment sys tems), and the VA to use in their efforts to improve the quality of care for people with substance use disorders. Currently, commercial health plans are being held accountable for improving their base line AOD services because performance measures have been included in the HEDIS national reporting system described earlier. States generally fund AOD treatment through Medicaid pro grams and through Substance Abuse Prevention and Treatment (SAPT) block grants. These block grants focus on State accountability by requiring States to measure current performance, set targets, and make changes to meet these targets for delivery of AOD services.
Certainly, many health plans and public payers have developed perfor mance measures focused on services for substance use disorders as part of their internal quality improvement initiatives. For example, the VA annually publishes the results of seven performance indica tors showing data both nationally and for specific facilities and networks (Harris et al. 2005). Oklahoma has developed a regional performance management report to offer quarterly feedback to providers, consumers, department administrators, and other stakeholders on key indicators of the performance of AOD and mental health treatment providers (Oklahoma Department of Mental Health and Substance Abuse Services 2005).
This article has focused on perfor mance measures that are being developed at a national level for widespread use. The three Washington Circle measures described, although developed to be applied at the health-plan level, also may be useful for quality improvement efforts at the local or facility level. For example, to pinpoint areas for improve ment, health plans might break down information on initiation and engage ment by age, gender, provider group, location, type of substance used, employ ment status (i.e., employee or depen dent), or initial care by specialty or general medical providers. States might provide prompt feedback on initiation and engagement rates to allow facilities to track improvement efforts.
The development of performance measures for AOD services still is at an early stage, and ongoing efforts are needed to create measures that encom pass all four domains--prevention/ education, recognition, treatment, and maintenance--originally envisioned by the Washington Circle. In addition, ongoing resources are required to do the following: • Conduct research to link process measures explicitly to outcomes.
• Update measures as new evidence on treatment effectiveness emerges and new ways of conceptualizing addiction are developed.
• Monitor the opportunities afforded by ongoing efforts for the improve ment of administrative data systems, linkage across multiple sources of services, and new electronic medical records.
• Disseminate measures that are meaningful to frontline providers (Valenstein et al. 2004).
As new data sources such as electronic medical records are used more com monly, it may become feasible to com bine performance measure concepts with evidence-based practice guidelines that are more focused on specific treat ment and allow for provider discretion (Garber 2005). For example, practice guidelines may include information on selecting an appropriate referral for a patient's clinical condition, whereas performance measures indicate whether the patient received followup services but do not give details about whether it was the right service. SAMHSA's new set of National Outcome Measures focused on AOD treatment and prevention is designed to facilitate achievement of the overarching goal of providing reli able and valid assessment of important dimensions of service system performance that can be useful at the national, State, and local levels (SAMHSA 2005b).
Performance measures are a starting point in quality improvement for people who do not receive adequate treatment for their addiction problems. These measures can be powerful tools for drawing attention to deficits in the current treatment system, monitoring the effectiveness of efforts to improve quality, designing incentives for quality improvement, and targeting areas where quality improvement is needed.